An evaluation of the inter-rater reliability in a clinical skills objective structured clinical examination

Background. An objective structured clinical examination (OSCE) is a performance-based used to assess health sciences students and awell-recognised tool skills with or without using real patients.Objectives. To determine the inter-rater reliability of experienced novice assessors from different backgrounds on final mark allocationsduring assessment third-year medical students’ OSCE at University Free State.Methods. This cross-sectional analytical study included 24 145 students. After training written instructions, two per station(urology history taking, respiratory gynaecology assessment) each independently assessed same student for skill bycompleting their individual checklists. At station, could also give global rating (from 1 5) as an overall impression.Results. The urology history-taking station had lowest mean score (53.4%) highest (71.1%). Seven (58.3%) ofthe 12 assessor pairs differed by >5% regarding mark, differences ranging 5.2% 12.2%. For entire confidence interval(CI) was within 5% range, whereas five CI outside range. Only one pair achieved substantial agreement (weightedkappa statistic 0.74 ‒ taking). There no consistency across stations whether assessorgave higher marks. stations, all majority globalrating mark. Weighted kappa statistics indicated that this mark.Conclusion. Despite previous experience, instructions in use checklists, between were found inmost cases.